<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Run Experiment</title>
    <link rel="stylesheet" href="doc.css">
  </head>
  <body>
<!--#include virtual="_header.html" -->


  <div id=content>
<h1>Run Experiment</h1>
<h2>Overview</h2>

This feature allows you to run experiments where segments of your traffic get
the page rewritten with different settings in order to figure out which filters
work best for your site.  It reports to your
<a href="http://www.google.com/analytics/">Google Analytics</a> account, storing
data in a
<a href="/analytics/devguides/collection/gajs/gaTrackingCustomVariables">custom
variable</a>
or <a href="https://developers.google.com/analytics/devguides/collection/analyticsjs/experiments">content
experiment</a>.

<h2>Configuration</h2>
<p>
To run an experiment you must set several options in the configuration file.
First, you turn on this feature and tell it
your <a href="http://support.google.com/analytics/bin/answer.py?hl=en&answer=1032385">Web
Property ID</a>:
</p>
<dl>
  <dt>Apache:<dd><pre class="prettyprint">
ModPagespeedRunExperiment on
ModPagespeedUseAnalyticsJs off  // use ga.js
ModPagespeedAnalyticsID UA-XXXXXXX-Y</pre>
  <dt>Nginx:<dd><pre class="prettyprint">
pagespeed RunExperiment on;
pagespeed UseAnalyticsJs off;
pagespeed AnalyticsID UA-XXXXXXX-Y;</pre>
</dl>
<p>
This will enable several filters needed for experiments including
the <a href="filter-insert-ga">Insert Google Analytics</a> filter to insert
the <code>ga.js</code> tracking snippet into each page.  If you already have a
tracking snippet on your pages, the lowest risk option is to remove it so that
the one PageSpeed inserts is the only one.  If you do choose to leave your
existing snippet, PageSpeed will attempt to modify it to add experiment
tracking.  If you've customized your snippets you should manually verify that
the modified snippet is still correct.
</p>
<p>
To disable experiments and experiment tracking you can set RunExperiment to
'off':
<dl>
  <dt>Apache:<dd><pre class="prettyprint"
     >ModPagespeedRunExperiment off</pre>
  <dt>Nginx:<dd><pre class="prettyprint"
     >pagespeed RunExperiment off;</pre>
</dl>
If you still want PageSpeed to automatically insert the Google Analytics
tracking snippet, you need to enable the insert_ga filter:
<dl>
  <dt>Apache:<dd><pre class="prettyprint"
     >ModPagespeedEnableFilters insert_ga</pre>
  <dt>Nginx:<dd><pre class="prettyprint"
     >pagespeed EnableFilters insert_ga;</pre>
</dl>
</p>
<p>
Once you have turned on RunExperiment and set your Google Analytics id you can
set up an experiment.  For example, to test how much PageSpeed is speeding
up your site you can apply optimizations for only half your visitors:
</p>
<dl>
  <dt>Apache:<dd><pre class="prettyprint">
ModPagespeedExperimentSpec id=1;percent=50;default
ModPagespeedExperimentSpec id=2;percent=50</pre>
  <dt>Nginx:<dd><pre class="prettyprint">
pagespeed ExperimentSpec "id=1;percent=50;default";
pagespeed ExperimentSpec "id=2;percent=50";</pre>
</dl>
<p>
Each <code>ExperimentSpec</code> defines an experimental treatment or
specification.  We only insert experiment tracking for users that fall into a
defined group, so you should define both an experiment group and a control
group.  The specification is a semicolon-separated list of settings.  Some
settings are required:
<dl>
  <dt><code>id</code><dd>A positive integer, unique across all experiments.  You
      can't reuse ids from one experiment to the next; every id must be unique.
      If you stop running an experiment that had three experiment specs 1, 2,
      and 3, then your next experiment should start with id 4 or higher.
  <dt><code>percent</code><dd>An integer between 0 and 100 indicating what
      fraction of users should receive this treatment.  If the sum
      of <code>percent</code> over all treatments is less than 100 then the
      remaining users will be assigned to 'no experiment' and continue seeing
      what they usually would.  If the sum exceeds 100 it will report an error
      when parsing the config file.
</dl>
</p>
<p>
ExperimentSpec 1 above adds the optional setting <code>default</code>, described
below, which tells PageSpeed to apply just the filters and settings
it would normally apply.  ExperimentSpec 2 specifies nothing and so turns off
all filters.
</p>
<p>
To report on these settings, PageSpeed will inject JavaScript into the page
to send data to your Analytics account.  It will set one of the following for
each visitor:
<pre class="prettyprint">
_gaq.push(['_setCustomVar', 1, 'ExperimentState', 'Experiment: 1']);
_gaq.push(['_setCustomVar', 1, 'ExperimentState', 'Experiment: 2']);
</pre>
</p>
<p>
In addition to <code>id</code> and <code>percent</code> described above, there
are additional optional settings intended to give you a way to test many
settings you could set in the configuration file:
</p>
<dl>
  <dt><code>default</code><dd>Apply the filters and settings that would
      normally apply.  Warning: prior to 1.9.32.1 this setting could not be
      used in combination with any others.  It would override any other
      configuration specified in the experiment spec, and the other settings
      would be silently ignored. This meant that
      in <code>default,enabled=remove_comments</code>
      the <code>enabled=remove_comments</code> would have no effect.  As of
      1.9.32.1, however, it is now safe to combine <code>default</code>
      with <code>enabled=</code>, <code>disabled=</code>,
      or <code>options=</code>, and it indicates that the current non-experiment
      options should be used as a starting point.  For example,
      in <code>default,enabled=remove_comments</code>, you would now get the
      expected result of server default filters and options,
      plus <code>remove_comments</code>.  Note that <code>level=</code> may
      still not be used with <code>default</code>.
      <p><br>
      While all the other settings are in the form <code>key=value</code>, as
      in <code>percent=42</code>, this setting has no 'value' and is
      just <code>default</code>.
  <dt><code>level</code><dd>Set
      the <code><a href="config_filters#level">RewriteLevel</a></code>.
  <dt><code>enabled</code><dd>A comma-separated list of filters explicitly
      enabled.  For example, <code>enabled=rewrite_images,inline_js</code>.
      Equivalent
      to <code><a href="config_filters#enabling">EnableFilters</a></code>.
  <dt><code>disabled</code><dd>A comma-separated list of filters explicitly
      disabled.  Equivalent
      to <code><a href="config_filters#enabling">DisableFilters</a></code>.
  <dt><code>options</code><dd>
      A comma-separated list of options to set.  For example,
      <tt>options=JpegNumProgressiveScans=5,WebpRecompressionQuality=72</tt>.
  <dt><code>ga_id</code><dd>The Google Analytics ID you would like the data of
      this experiment reported to.  If not specified, this experiment will use
      the Analytics ID specified with
      <code><a href="filter-insert-ga">AnalyticsID</a></code>.
  <dt><code>slot</code><dd>Google Analytics provides five slots for custom
      variables.  Here you specify which custom variable slot you would
      like to use.  The default is slot 1, but you can change this setting
      globally with:
      <pre class="prettyprint">ExperimentVariable 2</pre>
</dl>
<p>
Visitors are assigned to ExperimentSpecs on a week-to-week basis.  This
means that when you finish one experiment and start another, it will take up to
a week for repeat visitors to be assigned to the new experiment.
</p>
<p>
Starting in 1.8.31.2 you can test experiment configurations on a live site
before assigning anyone to them by creating them initially with a percentage of
'0%' and then visiting the page
with <code>?PageSpeedEnrollExperiment=&lt;experiment_id&gt;</code>.  This will
set a cookie that assigns you to the group you're trying to test.  You can stop
running this test by deleting the <code>PageSpeedExperiment</code> cookie or
loading the page with <code>?PageSpeedEnrollExperiment=-1</code>.
</p>
<h2>Examples</h2>
<p>
Run an experiment on 30% of visitors where half get the default configuration
and half get no filters.  Use Google Analytics ID UA-XXXXXXX-Y to run the
experiment, and default to logging the experiment information into custom
variable slot 1:
</p>
<dl>
  <dt>Apache:<dd><pre class="prettyprint">
ModPagespeedRunExperiment on
ModPagespeedAnalyticsID UA-XXXXXXX-Y
ModPagespeedExperimentSpec id=1;percent=15;default
ModPagespeedExperimentSpec id=2;percent=15</pre>
  <dt>Nginx:<dd><pre class="prettyprint">
pagespeed RunExperiment on;
pagespeed AnalyticsID UA-XXXXXXX-Y;
pagespeed ExperimentSpec "id=1;percent=15;default";
pagespeed ExperimentSpec "id=2;percent=15";</pre>
</dl>
<p>
Run an experiment on 30% of visitors where:
<ul>
<li>One tenth of visitors get the default configuration.
<li>One tenth get a new configuration which has the core filters
plus <code><a href="reference-image-optimize#inline_preview_images"
>inline_preview_images</a></code> and <code><a href="filter-comment-remove"
>remove_comments</a></code> without <code><a href="filter-js-minify"
>rewrite_javascript</a></code>, with <a href="restricting_urls#aris"
>AvoidRenamingIntrospectiveJavascript</a> disabled, and a CSS inlining limit of
4kB.
<li>One tenth get no filters.
<li>The remaining 70% aren't included in the experiment.
</ul>
All the other visitors will be unaffected.  For measurement it logs to custom
variable slot 3 of Google Analytics account UA-XXXXXXX-Y.
</p>
<dl>
  <dt>Apache:<dd><pre class="prettyprint">
ModPagespeedRunExperiment on
ModPagespeedAnalyticsID UA-XXXXXXX-Y
ModPagespeedExperimentVariable 3
ModPagespeedExperimentSpec id=3;percent=10;default
ModPagespeedExperimentSpec id=4;percent=10;level=CoreFilters;enabled=inline_preview_images,remove_comments;disabled=rewrite_javascript;options=AvoidRenamingIntrospectiveJavascript=off,CssInlineMaxBytes=4096
ModPagespeedExperimentSpec id=5;percent=10</pre>
  <dt>Nginx:<dd><pre class="prettyprint">
pagespeed RunExperiment on;
pagespeed AnalyticsID UA-XXXXXXX-Y;
pagespeed ExperimentVariable 3;
pagespeed ExperimentSpec "id=3;percent=10;default";
pagespeed ExperimentSpec "id=4;percent=10;level=CoreFilters;enabled=inline_preview_images,remove_comments;disabled=rewrite_javascript;options=AvoidRenamingIntrospectiveJavascript=off,CssInlineMaxBytes=4096";
pagespeed ExperimentSpec "id=5;percent=10";</pre>
</dl>
<p>
While the RunExperiment feature supports testing many changes, as shown above,
the discipline of changing only one variable at a time helps you see the effect
and know where it's coming from.  The only drawback is that sometimes two
filters are more useful in combination, such as minification and inlining, where
minifying brings resources down to the inlining threshold.
</p>
<p>
There is also a tradeoff with the experiment percentages: a larger percentage
means you will collect sufficient data for meaningful results more quickly but
it also affects more of your users.  If you're
<a href="experiment.html">testing manually</a> with query parameters to be sure
your settings aren't making things worse for you, the downside of running with
large experiment percentages is low.  While you may not have determined the
ideal settings right away, you're unlikely to have made things much worse.
</p>

<h2>Reporting</h2>
<p>
After you have been running an experiment for 24 hours Analytics will have
results for you.  The screenshots below walk through the process of creating a
custom report to view them.
<p>
<ol>
<li><h3>Set up advanced segments</h3>
  <ol type="a">
  <li>Advanced segments let you label fractions of your traffic.  To view speed
      results broken down by experiments you need to add segments for
      each experimental group.  Click <a target="_blank"
      href="https://www.google.com/analytics/web/permalink?type=advanced_segment&amp;uid=16TeL7ueRGuGttAzoYeV3A">here</a>
      to import a segment for a sample experimental group.
  <li>If prompted, log into Google Analytics:
    <div><img class="experiment-screenshot"
              alt="Google Analytics login screen."
              src="analytics_screenshots/1_login.png"></div>
  <li>Select the profile that your experiment data was collected under, enter
      a name for the segment, and click <strong>Create Advanced
      Segment</strong>:
    <div><img class="experiment-screenshot"
              alt="Import advanced segment screen."
              src="analytics_screenshots/2_import_advanced_segment.png">
    </div>
  <li>On the next screen you can adjust this custom segment to fit your
      situation.  If you used a custom variable slot other than the default,
      which is 1, change <strong>Custom Variable (Key 1)</strong>
      and <strong>Custom Variable (Value 01)</strong> to the appropriate values.
      Where it says <strong>Experiment: 1</strong> replace 1 with the experiment
      id you used. You also need to change <strong>FuriousState</strong> (our
      initial code name) to <strong>ExperimentState</strong>.

    <div><img class="experiment-screenshot"
              alt="Advanced segment settings."
              src="analytics_screenshots/3_segment_settings.png">
    </div>
  <li>Click <strong>Save Segment</strong>.
  <li>Repeat these steps for the rest of your experiment ids.
  </ol>
<li><h3>Add advanced segments to the speed report</h3>
  <ol type="a">
  <li>Click <strong>All Accounts</strong>.
  <li>On the dropdown menu click the name for your profile.  In the screenshot
      below this is labeled <strong>your profile name [DEFAULT]</strong>.
    <div><img class="experiment-screenshot"
              alt="Profile switching dropdown."
              src="analytics_screenshots/4_profile_switch.png">
    </div>
  <li>Click <strong>Standard Reporting</strong>:
    <div><img class="experiment-screenshot"
              alt="After profile switching."
              src="analytics_screenshots/5_profile_switched.png">
    </div>
  <li>Click <strong>Content</strong>
    <div><img class="experiment-screenshot"
              alt="Standard reporting screen."
              src="analytics_screenshots/6_standard_reporting.png">
    </div>
  <li>Under <strong>Content</strong> click <strong>Site
      Speed</strong> and then <strong>Page Timings</strong> to bring up
      the <strong>Site Speed Page Timings</strong> page:
    <div><img class="experiment-screenshot"
              alt="Page timings screen."
              src="analytics_screenshots/7_page_timings.png">
    </div>
  <li>Click <strong>Advanced Segments</strong>.
  <li>Under <strong>Custom Segments</strong> you will see the segments you added
      above.  Check the boxes next to them and click <strong>Apply</strong>:
    <div><img class="experiment-screenshot"
              alt="Advanced segment selection screen."
              src="analytics_screenshots/8_advanced_segments.png">
    </div>
  <li>This takes you to back to the <strong>Site Speed Page Timings</strong>
      page.
  </ol>
<li><h3>Examine experimental results</h3>
  <ol type="a">
  <li>The <strong>Explorer</strong> tab shows average timing
      information both overall and, below, per-url:
    <div><img class="experiment-screenshot"
              alt="Explorer tab."
              src="analytics_screenshots/9_page_level_timings.png">
    </div>
  <li>Averages can be misleading, however, because one user who suffers
      abnormally high latency before the onload event will have an outsize
      effect on the average.  Unless your sample is very large, such users can
      make an experiment falsely appear to be a success or failure.  To overcome
      this problem a histogram is helpful.  Click
      the <strong>Performance</strong> tab to see page load times grouped into
      intervals.
    <div><img class="experiment-screenshot"
              alt="Performance tab."
              src="analytics_screenshots/10_histogram.png">
    </div>
  <li>To see more fine grained timing detail, click the <strong>+</strong> icon
      next to any histogram interval to expand it.
  <li>In this experiment PageSpeed appears to have moved about 8% of visits
      from the 1-3 second category to the 0-1 second one.  For a simple
      mostly-text site running only the core set of filters, this is a
      reasonable result.  As a next step the webmaster of this site might
      examine the <a href="config_filters">filter list</a> to determine if there
      are filters that are not in the core set which might be good targets for
      future experimentation.
  </ol>
</ol>

<h2 id="content-experiments"
  >Integration with Google Analytics Content Experiments</h2>
<p class="note"><strong>Note: New feature as of 1.10.33.0</strong></p>

<p>
The usage described above reports experiment results to a custom variable in
Google Analytics.  As of 1.10.33.0 PageSpeed supports reporting experiment
results to a GA Content Experiment.  To do this, set up a
<a href="https://developers.google.com/analytics/devguides/collection/analyticsjs/experiments#pro-server">server-side
content experiment</a>.  When setting this up GA will tell you an "Experiment
ID" (which PageSpeed calls <code>ContentExperimentID</code>) and, for each
variation, a "Chosen Variation Index" (which PageSpeed
calls <code>ContentExperimentVariantID</code>).  Tell PageSpeed about these via
the <code>options=</code> parameter in the <code>ExperimentSpec</code>:
</p>

<dl>
  <dt>Apache:<dd><pre class="prettyprint">
ModPagespeedExperimentSpec id=1;percent=15;options=ContentExperimentID=ID:ABC123,ContentExperimentVariantID=1;default
ModPagespeedExperimentSpec id=2;percent=15;options=ContentExperimentID=ID:ABC123,ContentExperimentVariantID=2</pre>
  <dt>Nginx:<dd><pre class="prettyprint">
pagespeed ExperimentSpec "id=1;percent=15;options=ContentExperimentID=ID:ABC123,ContentExperimentVariantID=1;default";
pagespeed ExperimentSpec "id=2;percent=15;options=ContentExperimentID=ID:ABC123,ContentExperimentVariantID=2";</pre>
</dl>

<p>
While you can use the Content Experiment integration with
either <code>ga.js</code> or <code>analytics.js</code> we recommend
using <code>analytics.js</code> if possible.  Not only is <code>ga.js</code>
obselete, with Content Experiments it requires PageSpeed to insert a blocking
script, which slows down your page.
</p>

<h2>Technical Implementation</h2>
<p>
When a visitor first arrives on a site with this feature enabled PageSpeed
chooses an experiment from among the ExperimentSpecs.  If you are running your
experiments on less than 100% of your visitors, some will be assigned to "no
experiment", which PageSpeed represents as <code>id=0</code>.  It generates the
page in accordance with the ExperimentSpec and then sets a cookie valid for one
week, so that on future page loads it knows which ExperimentSpec to apply to
this visitor:
</p>
<pre>
   Date: Thu, 10 May 2012 14:19:43 GMT
   Server: Apache/2.2.14 (Ubuntu)
   Accept-Ranges: bytes
   Set-Cookie: PageSpeedExperiment=3; Expires=Thu, 17 May 2013 14:19:43 GMT; Domain=.www.example.com; Path=/
</pre>
You can test this on your site with the <code>curl</code> program:
<pre>
   $ curl -D- -o /dev/null http://yoursite
   ...
   Set-Cookie: PageSpeedExperiment=1; ...
   ...
   X-Mod-Pagespeed: ...
   ...
</pre>
<p>
If you don't see a <code>Set-Cookie</code> header in the output, this feature
isn't set up properly.  If you don't see an <code>X-Mod-Pagespeed</code> header
(Apache) or an <code>X-Page-Speed</code> header (Nginx), PageSpeed isn't running
at all.  On each page PageSpeed adds some JavaScript to set a Google Analytics
custom variable:
</p>
<pre class="prettyprint">
   _gaq.push(['_setCustomVar', 1, 'ExperimentState', 'Experiment: 4']);
</pre>
<p>
While it does try to detect when a page already is using Google Analytics and
add to it only what it needs, this is imprecise and can get it wrong.  We
recommend that you let PageSpeed insert the only snippet.  If you need to
call <a href="https://developers.google.com/analytics/devguides/collection/gajs/methods/">other
tracking methods</a>, for example to set
the <a href="https://developers.google.com/analytics/devguides/collection/gajs/methods/gaJSApiBasicConfiguration#_gat.GA_Tracker_._setSampleRate">sample
rate</a> for your site, you can do this by adding to the <code>_gaq</code> array
within <code>&lt;head&gt;</code>.  The code to do this for setting the site
sampleRate to 80% would be:
<pre class="prettyprint">
   _gaq = _gaq || [];
   _gaq.push(['_setSampleRate', '80']);
</pre>
  </div>
  <!--#include virtual="_footer.html" -->
  </body>
</html>
